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ABSTRACT 



As part of the evaluation of schools participating in the 
Comprehensive School Reform Demonstration (CSRD) program, a site researcher 
is to visit each participating site twice a year to collect data for CSRD 
formative evaluation. To prepare site researchers for their evaluation 
activities, a training and orientation session was held for 18 prospective 
site researchers. This document reports on the evaluation of the site 
researcher training. A feedback form was used to assess the degree to which 
certain goals and objectives were met, and a modified participant observation 
technique was used to give presenters immediate feedback. Program materials 
were also reviewed. Participants found the training sessions to have been 
conducted in a professional manner by competent professionals, and they 
considered the sessions relevant to their needs. Participants seemed to have 
learned most about the role of formative evaluation in the CSRD program, 
comprehensive school reform, and specific data collection techniques. They 
felt the need for further information about comprehensive school reform 
models, identification of and collaboration with a local coordinator, and 
collaboration with assigned external facilitators. Recommendations are made 
to improve program presentation. Appendixes contain the training feedback 
form and a checklist for applying the standards from the "Program Evaluation 
Standards" (1994) . (SLD) 
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INTRODUCTION 



Comprehensive School Rciorm Demonstration 

The Comprehensive School Reform Demonstration (CSRD) program, new in 1998, 
provides $150 million for local schools to implement comprehensive school reform programs. 
Much of these CSRD funds flowed through a competitive grant process to schools and districts 
interested in implementing schoolwide, comprehensive reform. Of these monies, $120 million 
came from Elementary and Secondary Education Act (ESEA) Title I funds, allocated by formula 
to states to allow current Title I schools to adopt comprehensive school reform models. The 
Fund for the Improvement of Education (FIE) allocated $25 million to flow to states by school- 
age population to allow non-Title I schools to implement reform programs as well. In addition, 
$4 million was allocated to Regional Educational Laboratories to assist states, districts, and 
schools in the implementation of the CSRD program. The remaining $1 million was used by the 
U.S. Department of Education to inform states and local schools about existing comprehensive 
reform models. 

As part of the application process to receive funds, schools were required to plan for the 
evaluation of CSRD implementation. To assist in meeting this evaluation need in local schools, 
part of AEL’s CSRD work is to develop and deliver CSRD formative evaluation design materials 
and processes. To this end, AEL has undertaken a joint venture with the Center for Research in 
Educational Policy (CREP) at The University of Memphis to implement the Formative 
Evaluation Process for School Improvement (FEPSI). 
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Formative Evaluation Process for School Improvement 

The Formative Evaluation Process for School Improvement (FEPSI) is a package of 
services designed to facilitate schools’ formative evaluation of their comprehensive school 
reform initiatives. It is provided to CSRD schools through the joint AEL/CREP initiative at a 
reasonable cost. To carry out FEPSI and complete the appropriate instrumentation, a trained 
individual called a site researcher is assigned to each participating site. The site researcher has 
primary responsibility for school-level data collection for CSRD formative evaluation. She or he 
conducts two critical site visits per year in each assigned school, one in the fall and one in the 
spring. The fall and spring visits each consist of a different slate of data collection activities. 

During the fall visit, the site researcher conducts (1) a principal interview, (2) a School 
Observation Measure ™ (SOM) comprised of several classroom visits, and (3) a benchmark- 
writing session with a committee of faculty. The principal interview is a 1-hour forum for 
learning a principal’s perceptions of CSRD implementation in his or her school. The SOM, an 
instrument developed by CREP, is a summary of ten 15 -minute classroom observations typically 
conducted in the course of a few hours. In brief, the items on which classrooms are observed are 
best practices in instruction and instructional orientation that characterize the intended nature and 
organization of CSRD schools. Also adapted and designed by CREP, benchmarking is a process 
by which schools chart the course of CSRD implementation relative to the nine criteria; the 
resultant benchmark document is intended to be a signatory piece that a school owns and refers to 
over the course of three years of implementation and thereafter. 

In the intervening period between the two visits, schools receive technical assistance and 
suggestions for revision on the first draft of their benchmark document by phone and mail from 
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an AEL/CREP staff member. Also during this period, eight additional SOMs are conducted by a 
school’s external facilitator, which are added to the d?ua collected by the site researcher. 

During the spring visit, the site researcher conducts (1) another SOM, (2) a focus group 
with a random selection of teachers, (3) a benchmark review with the committee that wrote the 
benchmark document in the fall, (4) a whole-faculty meeting, and (5) an interview design 
process. 1 Similar to the principal interview, the focus group is a forum for learning teachers’ 
views of CSRD implementation. The benchmark review involves marking the school’s progress 
along a continuum expressed within the benchmark document. The purpose of the faculty 
meeting is to administer two questionnaires, the School Climate Inventory ™ (SCI) and the 
Comprehensive School Reform Teacher Questionnaire ™ (CSRTQ), both developed by the Center 
for Research in Educational Policy at The University of Memphis (AEL & CREP, 1999). The 
SCI assesses school climate along seven dimensions, consisting of 49 statements set to a 5-point 
Likert-type scale. The CSRTQ assesses a school’s movement toward school reform along four 
dimensions — professional development, resources, pedagogical change, and outcomes — and also 
its progress toward specific school benchmark goals. 

A procedure called interview design was incorporated into the FEPSI package as a way of 
gathering data from individuals from role groups other than the faculty. Interview design is a 
musical chairs way of collecting data from many individuals in a small amount of time. All 
participants engage in asking and answering questions and analyzing responses. Those to be 
included in the Interview Design process are community members, parents, and students. 



interview design is a technique used at the Wharton School, Management and Behavioral Science Center, 
introduced to AEL by Oralie McAfee (AEL & CREP, 1999a). 
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If it is possible to coordinate, it is desirable that a school’s external facilitator accompany 
the site researcher on both the fall and spring visits to a school. (External facilitators are 
individuals who have been assisting schools with the implementation of their CSRD efforts.) 
Following each visit, the site researcher is responsible for shipping data in appropriate formats to 
CREP. Based on the data collected by the site researcher and the external facilitator, staff of the 
AEL/CREP partnership prepare and mail a report to each participating school at the end of the 
year. Schools are assured of confidentiality and encouraged to use the report, along with their 
benchmarks, to adjust their activities to enhance CSRD implementation. 

Site Researcher Training and Orientation 

Site researchers each agreed to serve one to several schools (out of 101) in AEL’s four- 
state region (Kentucky, Tennessee, West Virginia, and Virginia) on a contractual basis. To 
prepare site researchers for their role in conducting FEPSI activities in these schools, a Site 
Researcher Training and Orientation for 1 8 prospective site researchers was held in Louisville, 
Kentucky, on September 13-15, 1999. A supplementary handbook compiled in a three-ring 
binder was given to each participant at the commencement of the training. It included overviews 
of CSRD and FEPSI, the scope of work for site researchers, sections explaining the various 
FEPSI activities, and some of the protocols to be used. An additional resource was provided 
during the instruction on SOMs, i.e., a booklet of definitions for the practices to be observed 
during a SOM (CREP, 1999). 

Purpose and Objectives of Evaluation 

In an ongoing effort to improve its services and to satisfy its obligations to the U.S. 
Department of Education’s Office of Educational Research and Improvement, AEL evaluates all 
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of its regional educational laboratory work. Accordingly, evaluation of AEL’s CSRD work is 
undertaken to assess these services and their potential outcomes and impact. Questions related to 
basic reactions to and learning from the training were collected during the September 1999 event. 
Questions related to the impact of the training on participants’ behavior and results of their 
learning will be assessed in subsequent evaluation activities. The specific objectives of the 
evaluation of this particular event were to 

1 . assess the extent to which the Site Researcher Orientation and Training met its stated 
goals and objectives 

2. determine participants’ knowledge gain and further need for skills required by the role 
of site researcher 

Audiences for This Report 

The primary audiences for this report are the CSRD staff at AEL and the staff of the 
AEL/CREP partnership. Secondary audiences include other AEL staff and management, and 
researchers and evaluators interested in comprehensive school reform. 
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EVALUATION METHODS 



Overall Evaluation Design and Components 

Three techniques were used to evaluate the Site Researcher Orientation and Training 
session. First, a feedback form was used to assess the degree to which certain goals and 
objectives were met and to assess participants’ learning and further needs. Second, a modified 
participant observation technique was used to give presenters immediate feedback. Third, 
informal, unstructured interviews were conducted with AEL/CREP staff to collect impressions 
regarding the implementation of the FEPSI process. In addition, the AEL evaluator reviewed 
the training materials provided to participants and used in the training. A set of research-based 
guidelines serve to undergird the evaluation of training in the CSRD efforts. 

Theoretical Guidelines. In general, there are three major reasons to evaluate training: 

(1) to assure that it is meeting the outlined goals and objectives, (2) to decide whether to continue 
or discontinue training, and (3) to gain information on how to improve future training 
(Kirkpatrick, 1994). The two reasons for evaluating this training were (1) to assure that it is 
meeting the outlined goals and objectives and (2) to gain information on how to improve future 
training. 

Kirkpatrick model. The Kirkpatrick model (Kirkpatrick, 1 994) for evaluating training 
programs is used as an overarching schema to assess the effectiveness of CSRD training efforts. 
This model proposes evaluating training on four levels. 

• Reaction - defined as the assessment of how participants react to the training 

• Learning - defined as the extent to which participants change attitudes, improve 

knowledge, and/or increase skills as a result of attending the program 



• Behavior-defined as the extent to which change in behavior occurs because a participant 
has attended the training program (For change to occur, four elements need to be present: 
(1) a person must have a desire to change, (2) a person must know what to do and how to 
do it, (3) a person must work in the right climate, and (4) a person must be rewarded for 
change.) 

• Results-defmed as the final results that occur because the participants attended the 
program 

Data Collection Methods/Instruments 

The data collection methods used to evaluate the Site Researcher Orientation and 
Training were (1) a workshop feedback form, (2) a modified participant observation strategy by 
an AEL evaluator, and (3) informal, unstructured interviews with AEL/CREP staff. 

Feedback form. A feedback form (see Appendix A) was developed by CSRD evaluators 
and staff to collect participants’ reactions to the Site Researcher Orientation and Training. The 
items for the feedback form were created from information collected by AEL evaluators prior to 
the event. This information included (1) evaluation forms used at prior training of a like nature, 
(2) a draft agenda developed by AEL/CREP staff and presenters, and (3) written materials 
describing CSRD site researcher roles and responsibilities, and required knowledge, skills, and 
experience. The feedback form was divided into two sections. The first section assessed the 
extent to which the training met various general and specific objectives of training usually 
measured by AEL. Participants were asked to rate the extent to which the sessions met certain 
objectives on a scale of 1 (not at all) to 5 (very much). The first section assessed level 1 , 
reaction, of the Kirkpatrick model. The second section measured the knowledge gained by the 
participants with respect to site researcher responsibilities and essential knowledge and skills, as 
well as the participants’ further need for information regarding the roles, responsibilities, and 



essential knowledge and skills. Participants were asked to rate each item on a scale of 1 (none 
or not at all) to 5 ( very much or a lot). This section measured level 2, learning, of the 
Kirkpatrick model. 

The instrument was administered to the participants by the AEL evaluator on-site near the 
completion of the training. Only those participants who were to serve in the role of site 
researcher completed the feedback form. Level 3 (behavior) and level 4 (results) of the 
Kirkpatrick model will be assessed at later points during the site researchers’ experience in the 
schools. 

Modified participant observation. The purpose of the modified participant observation 
was to observe the training activities in the role of a process evaluator, to record the completion 
of those activities, and to provide daily feedback to the training leaders. The participant observer 
was an AEL evaluator who took notes, sat with and discussed the training with the various 
groups of participants throughout the training, and met informally with participants as they were 
available to provide feedback to the evaluator. Participants were made aware that the focus of 
the observation was on the training and not participants. 

Informal, unstructured interviews. The AEL evaluator met briefly with four 
AEL/CREP staff to gather impressions regarding the current training in particular and the 
implementation of the FEPSI process in general. The participants were informed that their 
responses would be anonymous and that the information would be used only to form general 
impressions about the implementation process to date. 
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Data Analysis Procedures 

Data analyses for this evaluation were completed by AEL evaluators at the AEL facilities 
in Charleston, West Virginia. The quantitative analyses were conducted using SPSS for 
Windows 9.0. Simple descriptive statistics were calculated for each item for the various 



participants. 



EVALUATION FINDINGS 



This section presents the findings from the data collection methods used in this 
evaluation. 

Workshop Evaluation Forms 

Eighteen participants completed the feedback form for the Site Researcher Training and 
Orientation sessions. The form was distributed by the AEL evaluator on-site near the end of the 
last day of the training. The AEL evaluator collected the forms as the participants completed 
them. 
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Table 1 displays the descriptive statistics for items in Section A of the feedback form. 



Table 1 

Descriptive Statistics for 
Section A of Training Feedback Form* 



Item 


N 


Mean 


SD 


1 . Had clear outcomes 


18 


4.4 


.62 


2. Included clear directions for activities 


18 


4.4 


.70 


3. Facilitated development of new skills 


18 


4.6 


.61 


4. Was conducted in an appealing manner 


18 


4.6 


.78 


5. Was conducted in a professional manner 


18 


4.8 


.38 


6. Was conducted by competent presenter(s) 


18 


4.8 


.38 


7. Had activities that were well sequenced 


18 


4.4 


.51 


8. Had activities that reinforced content 


18 


4.4 


.70 


9. Included appropriate examples 


18 


4.6 


.51 


10. Was relevant to my needs 


18 


4.8 


.43 


1 1 . Had meaningful involvement of participants 


18 


4.4 


.78 


12. Caused me to reflect on my practices 


18 


4.4 


.62 


13. Caused me to examine some of my attitudes 


18 


3.7 


1.24 


14. Stimulated me to want to use the concepts, skills, and/or 
materials presented 


18 


4.4 


.86 


15. Provided materials that will be useful to me in my future work 
as a CSRD site researcher 


18 


4.8 


.51 


16. Increased communication and collegiality with others 


18 


4.4 


.70 


1 7. Increased my understanding of formative evaluation 


18 


4.3 


1.02 



*Scale of 1 (not at all) to 5 (very much) 
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Those items rated the highest included the following: was conducted in a professional 
manner (mean of 4.8, SD of .38); was conducted by competent presenters ) (mean of 4.8, SD of 
.38); was relevant to my needs (mean of 4.8, SD of .43); and provided materials that will be 
useful to me in my future work as a CSRD site researcher (mean of 4.8, SD of .51). Nearly all 
items were rated very positively, with all items (except one) having a mean of at least 4.3 (on a 
five-point scale). The only item receiving a mean rating of less than 4.0 was caused me to reflect 
on my practices. This item had a mean of 3.7, with a rather large standard deviation of 1 .24. It 
may be that the item is unclear or not specific enough, or that the participants viewed the material 
as new learning, not material to stimulate reflections of personal or professional attitudes. 

Table 2 presents the descriptive statistics for section B of the feedback form. Participants 
were asked to rate each item (which represented skills or roles for which the site researcher 
would be responsible) on two dimensions: (1) knowledge that the participants gained during the 
session and (2) their further need for information regarding the item. Table 2 also includes a 
difference score between the two dimensions, which reflects the discrepancy between what 
participants learned and elements for which they have further need of information. 



Table 2 

Descriptive Statistics for 
Section B of Training Feedback Form* 



Item 


Knowledge Gain 


Further Need 


Diff. 

Score 


N 


Mean 


SD 


N 


Mean 


SD 


18. Understanding the role of formative 
evaluation in the CSRD program 


18 


4.3 


1.03 


18 


1.8 


1.0 


2.5 


19. Knowledge of schools and 
classrooms 


18 


3.3 


1.24 


18 


2.2 


1.10 


1.1 


20. Learning data collection techniques 


18 


4.0 


1.14 


18 


2.2 


1.31 


1.8 


21. Oral and written communication 
skills 


18 


2.5 


1.42 


18 


1.4 


.86 


1.1 


22. Facilitation skills 


18 


2.9 


1.26 


18 


1.9 


1.23 


1.0 


23. Skills in interpreting and analyzing 
qualitative and quantitative data 


18 


2.9 


1.40 


18 


2.1 


1.23 


0.8 


24. Knowledge about and 

understanding of Comprehensive 
School Reform 


18 


4.0 


1.14 


18 


1.9 


.87 


2.1 


25. Knowledge about Comprehensive 
School Reform models 


18 


3.1 


1.23 


18 


3.1 


1.26 


0.0 


26. Familiarity with common strategies 
associated with school reform 


18 


3.2 


1.31 


18 


2.2 


1.25 


1.0 


27. Classroom observation skills 


18 


3.5 


1.20 


18 


2.1 


1.16 


1.4 


28. Technology skills 


18 


1.9 


1.23 


18 


2.1 


1.30 


-0.2 


29. Setting up school site visits 


17 


3.1 


1.58 


17 


1.9 


1.34 


1.2 


30. Collaboration with assigned 
external facilitator 


17 


2.7 


1.57 


17 


2.9 


1.69 


-0.2 


3 1 . Identification of and collaboration 
with a local coordinator 


17 


2.2 


1.51 


17 


3.0 


1.77 


-0.8 



*Scale of 1 (none or not at all) to 5 (very much or a lot) 
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Participants rated their knowledge gain the highest regarding the role of formative evaluation 
in the CSRD program (mean of 4.3, SD of 1.03), data collection techniques (mean of 4.0, SD of 
1.14), and Comprehensive School Reform (mean of 4.0, SD of 1.14). The respondents also 
indicate that they had relatively little further need for instruction in these items (means of 1.8, 

1.9, and 1.9 respectively), suggesting that they had received more than adequate information 
regarding those items. Most other items had moderate ratings of knowledge gain (means of 2.2 - 
3.5). This may indicate that the participants came with a good bit of knowledge with regard to 
those items. Technology skills was the lowest rated of the knowledge gain items (mean of 1 .9, 
SD of 1.23), perhaps indicating that the session did not address these skills and/or that 
respondents found themselves to be rather competent in this area already. 

The data seem to indicate that the areas where participants had the greatest further need for 
information were knowledge about Comprehensive School Reform models (mean of 3.1, SD of 
1.26); identification of and collaboration with a local coordinator (mean of 3.0, SD of 1.77); and 
collaboration with assigned external facilitator (mean of 2.9, SD of 1.69). This interpretation is 
also supported by the difference in scores (the relative discrepancy between knowledge gained 
and further need). The difference scores were 0 or negative for each of these three items. 

A reliability coefficient was calculated on the first 17 items of the feedback form to assess 
internal consistency. Other items on the form were not included in calculation of the reliability 
coefficient because they were independent items. Coefficient alpha was calculated to be r=.85. 
This indicates a relatively reliable instrument. 



14 




19 



Modified participant observation. The outputs of the modified participant observation 
technique were handwritten notes and mental impressions retained by the AEL evaluator during 
the training. These notes and impressions were shared with the training leaders on a regular 
basis throughout the day. In addition, they were shared with the workshop leaders during daily 
debriefing sessions. Formative evaluation feedback, including suggestions for improvements in 
the processes, positive comments by participants, and impressions of activities that went well, 
was provided in the debriefing sessions, one purpose being to help plan for the next day. 

Some of the suggestions for improvement from the participants included keeping to the time 
schedule on the agenda, possibly sending mailings to participants prior to the training, providing 
in the training materials the overheads used in the presentations, and allowing more time to 
practice and discuss the benchmarking process. The many positive comments from the 
participants related to the enthusiasm and expertise of the presenters, the usefulness of the SOM 
manual, and the use of an inspirational poem by the workshop leader near the end of the training 
session. 

Informal, unstructured interviews. The AEL evaluator spoke informally to a number of 
AEL and CREP staff during breaks from the training as a means of collecting information about 
how the FEPSI process would be implemented and how the AEL/CREP partnership was 
progressing. It was clear from the discussions with the various staff that the FEPSI process was a 
large undertaking. In addition, some staff were new to their positions and only beginning to 
define their roles and responsibilities. Though the unstructured interviews were brief and 
informal, a few themes emerged that are worth noting. 
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First, it appeared that the number of schools involved exceeded expectations, overburdening 
staff in preparing for the Site Researcher Orientation and Training. Second, there seemed to be 
some difficulty in identifying and recruiting site researchers. Third, clarification of AEL and 
CREP staff roles and responsibilities was needed to facilitate the work efforts of the AEL/CREP 



partnership. 



CONCLUSIONS 



Participants found the Site Researcher Orientation and Training sessions to have been 
conducted in a professional manner by competent presenters. 

Participants found the Site Researcher Orientation and Training sessions to be relevant to 
their needs and to have facilitated development of new skills, while providing materials 
useful to them in their future work as a site researcher for the Formative Evaluation Process 
for School Improvement (FEPSI) in CSRD schools. 

Participants seemed to have learned most about the role of formative evaluation in the CSRD 
program, comprehensive school reform, and specific data collection techniques associated 
with FEPSI. 

Participants seem to have the most need for further information regarding comprehensive 
school reform models, identification of and collaboration with a local coordinator, and 
collaboration with assigned external facilitators. 

More time for discussion of and practice with the benchmarking process might be useful. 
Participants found the actual hands-on experience with the School Observation Measure 
(SOM) and the benchmarking process very helpful. 

Clarification of specific roles and responsibilities of AEL and CREP staff related to the 
coordination of FEPSI might prove helpful to the ongoing implementation of FEPSI in 



AEL’s region. 



RECOMMENDATIONS 



• Include all overheads used by presenters in participants’ materials. 

• Consider providing more information regarding comprehensive school reform models in 
subsequent training materials. 

• Consider shipping materials to participants prior to the actual on-site training so that 
participants might complete any necessary reading and become more familiar with the 
materials/instruments prior to the training event. 

• If possible, have actual on-line demonstrations of the FEPSI activity management system. 

• Attempt to clarify roles and responsibilities of AEL/CREP partnership staff regarding 
ongoing implementation of FEPSI. 

• Consider allowing more time for the discussion of and practice with the benchmarking 
process. 

• Consider developing a more detailed checklist (incorporating elements of tabs 4 and 5 of 
current materials) of site researcher responsibilities that also includes a glossary of acronyms 
and terms. 
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Appendix A 

Training Feedback Form 



Site Researcher Orientation and Training 
September 13-15, 1999 
Louisville, Kentucky 

Feedback Form: Formative Evaluation Process for Scho ol Impr ovement (FEPSil 
Site Researcher ID# 

We are interested in continuously improving our services. Please help us to do so by taking a 
few moments to respond to the following items. Your replies will remain anonymous and 
confidential, so feel free to answer candidly. Thank you! 

Section A: Circle the number that best indicates the extent to which the session(s) on the 



Formative Evaluation Process for School Improvement (FEPSI) . . . 

Not at 
all 

1. Had clear outcomes ' 1 2 


3' 


4 


Very 

much 

5 


2. 


Included clear directions for activities 


1 


2 


3 


4 


5 


3. 


Facilitated development of new skills 


1 


2 


.3 


4 


5 


4. 


Was conducted in an appealing manner 


1 


2 


3 


4 


5 


5. 


Was conducted in a professional manner 


1 


2 


3 


4 


5 


6. 


Was conducted by competent presenter(s) 


1 


2 


3 


4 


5 


7. 


Had activities that were well sequenced 


1 


2 


3 


4 


5 


8. 


Had activities that reinforced the content 


1 


2 


3 


4 


5 


9. 


Included appropriate examples 


1 


2 


3 


4 


5 


10. 


Was relevant to my needs 


1 


2 


3 


4 


5 


11. 


Had meaningful involvement of participants 


1 


2 


3 


4 


5 


12. 


Caused me to reflect on my practices 


1 


2 


3 


4 


5 : 


13. 


Caused me to examine some of my attitudes 


1 


2.. 


3 


4 


5 


14. 


Stimulated me to want to use the concepts, 
skills, and/or materials presented 


1 


2 


3 


4 


5 


15. 


Provided materials that will be useful to me in 
my future work as a CSRD site researcher 


1 


2 


3 


4 


5 


16. 


Increased communication and collegiality 
with others 


1 


2 


3 


4 


5 


17. 


Increased my understanding of formative evaluation! 


2 


3 


4 


5 



Section B. Listed below are skills that you need and/or roles that you will be playing as a site 
researcher. You are asked to rate each item on two dimensions: 1 ) knowledge you gainea during 
the session, and 2) your need for further information regarding the item. Scale : 1 = None or not 
at all to 5 = Very much or a lot 



18. Understanding the role of formative 
evaluation in the CSRD program 

1 9. Knowledge of schools and classrooms 

20. Learning data collection techniques 

2 1 . Oral and written communication skills 

22. Facilitation skills 

23. Skills in interpreting and analyzing 
qualitative and quantitative data 

24. Knowledge about and understanding of 
Comprehensive School Reform (e.g., 
relationship to Title I, school-wide 
programs, Obey-Porter legislation) 

25. Knowledge about Comprehensive 
School Reform models 

26. Familiarity with common strategies 
associated with school reform v e.g., use 
of technology, cooperative and team- 
based strategies, student-centered learning) 

27. Classroom observation skills (e.g., 
recognizing and identifying what is 
happening in a class) 

28. Technology skills (e.g., e-mail capability, 
Internet access, data entry) 



Knowledge gain 
1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

12 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



Further need 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

12 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



29. Setting up school site visits 

30. Collaboration with assigned external 
facilitator 

3 1 . Identification of and collaboration 
with a local coordinator 



12345 12345 

12345 12345 

1 2 3 4 5 
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Appendix B 

Completed Checklist for Applying the Standards 
from The Program Evaluation Standards (1994) 



Checklist for Applying the Standards 



To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check a r , appropriate): 



Descriptor 

U 1 Stakeholder Identification 


The Standard was 
addressed 

X 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


U2 Evaluator Credibility 


x 








U3 Information Scope and Selection 


X 








U4 Values Identification 


X 








U5 Report Clarity 


X 








U6 Report Timeliness and Dissemination 




X 






U7 Evaluation Impact 


X 








FI Practical Procedures 


X 








F2 Political Viability 


X 








F3 Cost Effectiveness 


X 








P 1 Service Orientation 


X 








P2 Formal Agreements 




X 






P3 Rights of Human Subjects 


X 








P4 Human Interactions 


X 








P5 Complete and Fair Assessment 


X 








P6 Disclosure of Findings 


X 








P7 Conflict of Interest 


X 








P8 Fiscal Responsibility 


X 








A 1 Program Documentation 


X 








A2 Context Analysis 


X 








A3 Described Purposes and Procedures 


X 








A4 Defensible Information Sources 


X 








A5 Valid Information 


X 








A6 Reliable Information 


X 








A7 Systematic Information 


X 








A8 Analysis of Quantitative Information 


X 








A9 Analysis of Qualitative Information 


X 








A 10 Justified Conclusions 


X 








All Impartial Reporting 


X 








A 12 Metaevaluation 


X 









The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 



request for evaluation plan/design/proposal 

evaluation plan/design/proposal 

evaluation contract 

X evaluation report 

other: 




Address - 



P.0. Box 1348 Charleston, WV 25325-1348 



Relation to Document 



Primary author/Evaluation team leader 



(e.g., author of document, evaluation team leader, external auditor, internal auditor) 



erJc, best copy available 

iimnaffamiaaa 





TM031893 



U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reproduction Basis 




This document is covered by a signed "Reproduction Release 
(Blanket)" form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a "Specific Document" Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either "Specific Document" or "Blanket"). 
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